Word and Document Embeddings based on Neural Network Approaches

Data representation is a fundamental task in machine learning. The representation of data affects the performance of the whole machine learning system. In a long history, the representation of data is done by feature engineering, and researchers aim at designing better features for specific tasks. Recently, the rapid development of deep learning and representation learning has brought new inspiration to various domains. In natural language processing, the most widely used feature representation is the Bag-of-Words model. This model has the data sparsity problem and cannot keep the word order information. Other features such as part-of-speech tagging or more complex syntax features can only fit for specific tasks inmost cases. This thesis focuses onword representation and document representation. We compare the existing systems and present our new model. First, for generating word embeddings, we make comprehensive comparisons among existing word embedding models. In terms of theory, we figure out the relationship between the two most important models, i.e., Skip-gram and GloVe. In our experiments, we analyze three key points in generating word embeddings, including the model construction, the training corpus and parameter design. We evaluate word embeddings with three types of tasks, and we argue that they cover the existing use of word embeddings. Through theory and practical experiments, we present some guidelines for how to generate a good word embedding. Second, in Chinese character or word representation, we find that the existing models always use theword embeddingmodels directly. We introduce the joint training of Chinese character and word. This method incorporates the context words into the representation space of a Chinese character, which leads to a better representation of Chinese characters and words. In the tasks of Chinese character segmentation and document classification, the joint training outperforms the existing methods that train characters or words with traditional word embedding algorithms. Third, for document representation, we analyze the existing document representation models, including recursive neural networks, recurrent neural networks and coniv 基于神经网络的词和文档语义向量表示方法研究 volutional neural networks. We point out the drawbacks of these models and present our newmodel, the recurrent convolutional neural networks. In text classification task, the experimental results show that our model outperforms the existing models.